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STUDENT EVALUATION OF TEAuiiNG 
Donald J. Veldman 



Research Methodology Monograph No. 10 
R&D Center for Teacher Education 
The University of Texas at Austin 

This instrument was derived from research with a technique called 
the Pupil Observation Survey Report (POSR), which was the subject of an 
earlier monograph in this series (RMK-2). Briefly summarized, the purpose 
of the SET is economical measurement of the five major aspects of classroom 
behavior previously identified in research with student teachers. The SET 
contains only 10 items and utilizes an optically-scanned answer sheet, 
which yields data for a computer program that summarizes class responses 
and prints verbal reports for the teachers or their supervisors. 

Summary of Research with the P0S_R 

The POSR was an outgrowth of rese rch by McClain and Bown (1961) 

with an experimental instrument developed ,j McClain (1961). A preliminary 
version of the POSR was used by Veldman and Peck (1964) to study interact! o 
of pupil and teacher sex as they influence pupil perceptions. The only in- 
teraction observed was with a group of items callec “Identification Model." 

During the 1961-62 academic year, data were collected from the 
pupils of 554 student teachers (7th through 12th grades) at the University 
of Texas at Austin, using the 38-item P^R fc. ... A factor analysis of 
these data identified five major dimensions of the "space" within which 
pupils implicitly locate their teachers. The five factors were labeled 

as follows. 

1. Friendly and Cheerful 

2. Knowledgeable and Poised 

3. Lively and Interesting 

4. Firm Control (Discipline) 

5. Non-Directive (Democratic Procedure) 
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As noted in the published report of this research (Veldman and 
Peck, 1963), the first three of these factors bear a remarkable similarity 
to Ryans' (1960) three "patterns" of adult-observed teacher behavior. 

Analyses reported in this initial study clearly demonstrated 
factorial invariance of the POSR structure across three semester sub- 
samples and across teacher sexes. Reliability coefficients derived from 
the data of 50 teachers with two classes were, respectively: .92, .72, .91, 
.81, and .89. Comparisons of male and female teacher means indicated that 
females were rated significantly higher only on factors 1 and 5. Correlations 
between POSR factors and the scales of two self-report personality-atti- 
tude inventories were low, but v/ere frequently statistically significant 
and interpretable. Teaching effectiveness as rated by supervisors was signi- 
ficantly related to factor 1 (Friendly and Cheerful) among females only, 
and to factors 2 (Knowledgeable and Poised) and 4* (Strict Control ) among 
teachers of both sexes. 

A later research report by Veldman and Peck (1969) used complex 
covariance analyses to determine the degree to which pupil evaluations of 
student teachers (N=609) were related to (1) supervisor evaluations (factors 
1, 2, 3 were), (2) grade level of the class (factors 1 and 3 v/ere), (3) sub- 
ject matter area (factors 1, 3, 4, 5 were), (4) socioeconomic level of the 
school (factors 3 and 4 were), and (5) sex of the student teacher (only 
factor 1 was). The small absolute sizes of most of the effects led to the 
conclusion that pupil reports of teacher behavior are relatively free of 
bias. An exception, however, is the use of the POSR with physical educa- 
tion classes. 

The most recent report of research employing the POSR (Veldman, 
in press) concerned comparisons of pupil perceptions of 55 studenc teachers 
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and of their public-school teaching supervisors. The supervisors were 
seen as less friendly and cheerful, less lively and interesting, and less 
directive than the student teachers; but they were considered to be more 
poised and knowledgeable and more firmly controlling. Correlational evi- 
dence suggested that supervisors influenced the evaluations of the student 
teachers only with regard to factors 4 and 5, which suggested that the 
supervisors "set” the classroom atmosphere and routine before the student 
teachers arrive. 



Development of the SET 

All research carried out with data from the POSR made use of 
factor scores for teachers which were derived through the use of regression 
weights for all 38 items. In anticipation of the possibility that the 
simple sums of two or three items might be as useful as the regression- 

weighted factor scores, the three items loading each of the five factors 
most strongly in the original analysis were selected and simple sums of 
the class means of these items were computed for each of 562 student teachers. 
Table 1 shows the correlations between the regression-weig^ r 
scores and these simple sum scores. 

Table 1. Correlations of Sum Scores with Factor Scores 

(N = 562 student teachers) 



POSR Factor 2 items 3 items 



1. Friendly and Cheerful .91 .93 

2. Knowledgeable and Poised .87 .87 

3. Lively and INteresting .77 .79 

4. Firm Control .91 .92 

5. Non-Directive .78 .74 
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The use of three items rather than two affords no better estimation 
of the factor scores. The fact that some of these correlations are far 
from perfect does not necessarily indicate that they would be less valid 
or reliable meast/es. Schweiker (1967) makes a convincing case for the 
simpler measures, noting that they are easier to compute, are more directly 
interpretable, and are also less subject to distortion when the equations 
are applied to data not included in the original analysis. In at least 
one study (Veldman and Parker, 1970) regression-weighted factor scores have 
been shown to yield lower concurrent validity than comparable Likert-scale 

scores. 

Table 2 contains the ten items (two per factor) which were selected 
as the best estimators of the POSR factors, along with their slightly re- 
worded counterparts in the SET form. 

Table 2. POSR-SET It~m Or;, i -ce 

Factor I: Friendly and Cheerful 

SET-1: This teacher is always friendly toward students. 

P0SR-26: She is always friendly toward her students. 

SET-6: This teacher is usually cheerful and optimistic. 

POSR-29: She always seems cheerful and happy. 

Factor II: Knowledgeable and Poised 

SET-2: This teacher knows a lot about the subject. 

POSR-17: She knows a great deal about her subject. 

SET-7: This teacher is not confused by unexpected questions. 

POSR-34: She doesn't get confused by unexpected questions. 

r e 
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Factor III: Lively and Interesting 

SET-3: This teacher is never dull or boring. 

POSR-12: Her class is never dull or boring. 

SET-8: This teacher makes learning more like fun than work. 

POSR-31: She makes learning seem more like fun than work. 

Factor IV: Firm Control 

SET- 4: This teacher expects a lot from students. 

POSR-3: She expects a lot from her students and usually gets it. 

SET-9: This teacher doesn't let students get away with anything. 

POSR-32: She doesn't let her students get away with anything. 

Factor V: Non-Directive 

SET-5: This teacher asks for students' opinions before making 

decisions. 

POSR-8: Before she decides on a new project, she often asks students 

what they think. 

SET- 10: This teacher often gives students a choice in assignments. 

POSR-28: She likes to give the student a choice of how to do an 
assignment. 

On a following page is a copy of the optical -scanning answer sheet 
whi ch is the new SET form. The "true-false" four-choice response format 
of the POSR has been retained. Much of the rewording Vfas done to avoid the 
use of sex-specific pronouns in the new form. 

Comparison of POSR and Simulated SET Factor Structures^ 

To confirm the correspondence of the SET 10-item factor structure 
with the original POSR structure, the 10 selected POSR items were factored 
using the sample of 562 student teachers. Five factors were extracted and 
rotated by the varimax method. Table 3 shows the resulting structure. 
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STUDENT EVALUATION of TEACHING 

D.J. VELDMAN and R. F. PECK 



A. TEACHER’S LAST NAME:. 

B. SUBJECT: 

C. SCHOOL: 



INSTRUCTIONS: 

1. USE A NO. 2 PENCIL ONLY. 

2. PRINT THE INFORMATION FORA, B, AND C. 

3. MARK THE RIGHT BOXES FOR D, E, AND F. 

4. DO NOT MAKE ANY EXTRA MARKS. 

5. ERASE ERRORS COMPLETELY. 



D. 

E. 

F. 



MARK THE RIGHT BOXES FOR 
D, E and F BELOW 

TEACHER'S SEX: 

MY SEX: 

MY GRADE LEVEL: 

i 4' 5 5 V B 2 TO Vi 12 ;S 14 15 16 




MARK ONE OF THE FOUR BOXES IN FRONT OF EACH STATEMENT 
THE FOUR CHOICES ME AN: 

F - VERY MUCH FALSE 
f = MORE FALSE THAN TRUE 
t = MORE TRUE THAN FALSE 
= VERY MUCH TRUE 



£ 1 *■ * 

F Y « T 

.-,.J L- v-' 

l 3 3 i 



n n n n 

F f t T 

u Li la u 



F 7 7 T 

u U U 'Ll 



fi Z ‘ 5 

n ri n n 

F f t T 

l.s 1.; i..: 

v n 

F f I T 

U i- ’-I 

? ‘( ! ■; t 

u U U U 

F Y t T 



THIS TEACHER: 

is always friendly toward students, 
knows a lot about the subject, 
is never dull or boring, 
expects a lot from students. 

asks for students’ opinions betore making decisions, 
is usually cheerful and optimistic, 
is not confused by unexpected questions, 
makes learning more like fun than work, 
doesn't let students get away with anything, 
often gives students a choice in assignments. 
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Table 3. Varimax structure of 10 selected POSR Items. 



POSR 


SET 


POSR 






Factor 






Factor 


Item 


Item 


1 


2 


3 


4 


5 


1 


1 


26 


.89* 


.18 


.18 


.02 


.22 


1 


6 


29 


.89* 


.12 


.23 


.08 


.23 


2 


2 


17 


.19 


.89* 


.10 


.19 


.11 


2 


7 


34 


.11 


.84* 


.28 


.19 


.18 


3 


3 


12 


.39 


.33 


.71* 


.26 


,24 


3 


8 


31 


.45 


.28 


.70* 


.12 


. 36 


4 


4 


3 


.21 


.28 


-.06 


.86* 


.09 


4 


9 


32 


-.16 


.10 


.34 


.83* 


.04 


5 


5 


8 


.17 


.09 


.12 


.10 


.91* 


5 


10 


28 


.27 


o 

CM 

• 


.26 


.00 


.81* 



♦expected primary loadings 

It is obvious from the loadings in Table 3 that the expected primary 
loadings were, indeed, appropriately high. A further check was carried out 
using Program RELATE (Veldman, 1967) to re-rotate the varimax structure 
toward a hypothesis structure where every item was totally identified with 
one of the five factors. This technique is fully explained in another 
monograph of this series (RMM-8). All five correspondence coefficients for 
the factors exceeded ,98, and no pair of item vectors had a coefficient 
less than .82. In 'summary, the SET items yield a factor structure closely 

O approximating that of the original 38 POSR items. 
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Further information about these 10 item pairs is provided in Table 4, 
which contains their means and sigmas, as well as the intra-pair item corre- 

lati ons . 



Table 4. Means, sigmas and intra-pair correlations of items. 



Factor 


Item 


Mean 


Siqma 


Correlation 


1 

1 


1 

6 


3.58 

3.50 


.33 

.37 


.86 


2 


2 


3.60 


.24 


.73 


2 


7 


3.19 


.32 




3 

3 


* 3 
8 


2.90 

2.95 


.46 

.47 


.85 


4 


4 


2.91 


.32 


*.56 


4 


9 


2.82 


.36 




5 


Cv 

%r 


2.69 


.50 


.71 


5 


10 


2.77 


.40 




Despite 


the clearly 


separate identity of the 


five pairs of items 


in Table 


3, the Likert- type sum 


scores are more strongly intercorre- 


than would be factor scores computed with regression weights. 


5 shows 


the degree 


of this intercorrelation 


among the SET two- item 


s cores » 


as well as 


their means 


and sigmas. 
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Table 5. Intercorrelations among SET scale scores.* 



Scale 


1 


2 


3 


4 


5 


1 


_ _ 


.42 


.67 


.19 


.55 


o 


.42 


— 


.59 


.44 


.39 


3 


.67 


.59 


— 


• 

4 * 

O 


.60 


4 


.19 


.44 


.40 


— 


.20 


5 


.55 


.39 


.60 


.20 


“ - “ 


Mean 


7.01 


6.79 


5.85 


5.72 


5.46 


Sigma 


.61 


.52 


.89 


.59 


.83 


* Sum 


of two i 


terns; max 


= 8.0, 


min = 2.0 





Before going on to consider the scoring of the SET, we should note 
that all data presented so far were obtained by simulati nq the SET with 
item data actually obtained from POSR protocols. At present, data are 
not available from actual use of the SET instrument. 



Scoring the SET 

The FORTRAN program used to score SET protocols and to print summary 
reports may be found in Appendix A, along with example input. An example 
of the output reports is located in Appendix B. 



All of the SET items are positive statements, and the responses 
are numerically coded for punching as follows: F=l, f=2, t s 3, T-4 

The punching of cards from pupil answer sheets may be accomplished with a 
keypunch machine, or by a Digitek optical scanner. The card format recom- 



mended is as follows: 




col 1-5: teacher code number 

col 6: teacher sex 

col 7-8: class grade level 

col 9: optional (school, subject, etc.) 

col 10: pupil sex 

col 21-30: SET i tern scores (1-4 or bl ahk) 
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The next stage of the processing is accomplished by a computer 
program such as the one in Appendix A. The first step is the computa- 
tion of means for each of the 10 items. These are computed from valid 
responses, which may vary in number from one to another item. Blanks or 
double-marks are ignored. The printed report begins with these means and 
their verbal interpretations. These values range from 100 to 400, since 

the item means are multiplied by 100 before printing. 

The item means are then paired to form the five scale scores 
(divided by two to maintain scaling consistency), which are also verbally 
interpreted. Finally, the mean of all 10 item means is computed to provide 
an overall index of pupil evaluation of the teacher. 

At present, the scoring and interpretation is carried out on an 
"absolute" basis; no attempt has been made to reflect normative expecta- 
tions based on a population of student teachers. Eventually, such exten- 
sions of the interpretations may be added to the program. 

Interpretation of SET Profiles, 

As is the case with most aspects of human behavior, too much of 
a good thing is undesirable - especially when other good things are sacri- 
ficed. In one of the research studies described earlier, those student 
teachers who were rated highest In overall effectiveness by their supervi- 
sors were rated only moderately high on factors 3 (Lively and Interesting) 

and 4 (Firm Control) by their pupils. 

An even more Important point is that the scores should be considered 

together as a profile or pattern. For instance, very high scores on noth 
factor 1 (Friendly and Cheerful) and factor 4 (Firm Control) suggest a tea- 
cher who is firm with pupils without being cold or punitive. However, a 
O h factor 1 score with a low factor 4 score suggests too much concern with 

ERIC C 12 
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"being liked", while the opposite combination suggests harshly rigid dis- 
cipline. 

The present scoring and interpretation program does not analyze 
this pattern aspect of the SET data, although research is underway to 
develop such automatic interpretive procedures. 

Uses of the SET in Teacher Education. 

It seems quite obvious that a student teacher would profit from 
systematic knowledge of how pupils react to her as she takes on this 
professional role. In many cases the SET results will only confirm what 
she already knows, while in others it may reveal something about her im- 
pact on children of which she had not been fully aware. 

Feedback of SET results is probably most effective in a context 
where both the student teacher and her supervisor study the profile of 
scores and discuss its implications in terms of their personal perceptions 
of the student's approach to the teaching role. Although no data are yet 
available to support the viewpoint, an experienced consultant may be nec- 
essary to avoid over-interpretation and unwarranted conclusions on the 
part of the student. With appropriate guidance the SET can certainly do 
no harm and may be crucially important to the maximum development of the 
potential of some students. 
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APPENDIX a. 



SCORING AND REPORT-GENERATOR PROGRAM FOR THE SET 



PROGRAM SET (CDC6600 FORTRAN) 

...LABEL PACK OF 26 CARDS PRECEDES DATA 

...INPUT CARD FORMAT CAN BE MODIFIED < STATEMENT 25) IF 

COL 1- 9 - TEACHER IDENTIFIC 'ON 

COL 11-20 = TEN ITEM SCORES II JR BLANK) 

...EACH CLASS PACK OF PUPIL CARDS END 1 WITH A BLANK, 



• • • 
• • 6 




EXTRA BLANK FOLLOWS THE LAST CLASS PACK . 

SET KP = 1 IN PROGRAM TO GET PUNCHED SCORES. 

SET KR - 1 IN PROGRAM TO GET PRINTED PACiE RE-ORTS. 
OUTPUT CARD FORMAT. 

...COL 1- 9 = TEACHER I DENT I F I CAT I 
...COL 10-12 = NUMBER OF PUPILS IN CL A '" r : 

...COL 21-50 = 10 ITEM MEANS (*100) 

...COL 56-70 = 5 SCALE MEANS (*100) 

...COL 76-80 = MEAN OF ALL ITEM MEANS *1003 



DIMENSION A ( 10) . B(5)» D(10), V(10i* _A(8*-^!» LB ( 

1 LD( 8 »6 ) » LE ( 3 ) 

DATA LE / 9H RATHER. 9H QUITE. PH VERY/ 

KP = 0 

ICR — i 

READ 5. LA. LB. LC. LD 
5 FORMAT ( 8 A 1 0 ) 

10 N = 0 

DO 15 I = 1.10 
15 A( I ) « V < I 1 = 0 
20 READ 25. ID. D 
25 FORMAT < A9 . 11X. 10F1) 

IF (ID *NE. 1H ) GO TO 30 
IF (N .EQ. 0) STOP 



GO TO 40 



30 N = N + 1 
IDX = ID 
DO 35 I = 1.10 

IF (D( I ) .GT, 0) V ( I ) 
35 A ( I ) = Ad ) + D ( I ) 

GO TO 20 
40 C = CV = 0 



DO 45 I = 1 .10 
IF (V(I> .GT. 0) 
C = C + A ( I ) 

IF (A(I) .GT. 

45 CONTINUE 



A ( I ) 



= Vd) + 



= A ( I ) / 

CV + 1 



1 



V( I 1 



* 100 



NECESSARY, 



8.10). LC (4.2.5 



0 ) CV 



SET PROGRAM (CONTINUED) 



0) B ( I ) = B ( I ) / 2 



50 

55 



60 



65 

70 



C = C / CV 
DO 50 I = 1*5 

B ( I > = A(I> + All +5) 

IF ( A ( I ) * A< l +5 ) .GT 

CONTINUE 

IF (KP .GT. 0) PUNCH 55* I DX * N* A* B* 
FORMAT IA9t 13* 8X* 10F3* 5X* 5F3* F5 ) 

IF CKR e EQ . 0) GO TO 10 
PRINT 60*1 DX *A*B*C*N 

FORMAT ( * 1 * * 14X »*SUMMARY OF STUDENT 

1 / / / 5 X * *T E ACHER IDENTIFICATION = * 

2 5 X » *RAW SCORES =** 10F4* 3X » 5F4* 

3 10X, *THE*. 13. * STUDENTS IN THIS 

DO 65 I = 1*10 

K = A ( I ) / 100 + 1 .499 

PRINT LA< 1 * K. ) * <LB(J* I ) * J - 1*6) 

PRINT 70 _ 

♦PAIRED ITEMS SUGGEST 



EVALUATION 
, A9 // 

3X » F4 /// 
CLASS SAID 



OF TEACHING* 



THAT IT IS*) 



0) GO TO 80 



75 

80 



85 



FORMAT <//5X* 

DO 80 I = 1*5 

IF ( B ( I ) .EQ. 

M = L = 1 

IF ( B ( I ) .LT. 250) M = 2 
K = ABS ( B ( I ) “ 250) 

IF (K .GT. 50) L = 2 
IF (K .GT. 100 ) L = 3 
PRINT 75* LE ( L ) • <LC<J*M*I)* 

FORMAT ( /A15 * 4A10) 

CONTINUE 
K “ C / 50 — 1 

PRINT 85* (LD(J*K)* J = 1*8) 

FORMAT (//* IN GENERAL* THIS 
GO TO 10 
END 



THAT THIS TEACHER IS*) 



J = 1.4) 



CLASS HAS ** 8A10) 










set label pack and example deck 



( /5X »*( UNANSWERED) THAT THIS TEACHER **6A10> 

</5X**VERY FALSE THAT THIS TEACHER *.6A10> 

</5X**RATHER FALSE THAT THIS TEACHER ** 6A10 ) 

(/5X»*RATHER TRUE THAT THIS TEACHER **6A10) 

(/5X**VERY TRUE THAT THIS TEACHER **6A10) 

IS ALWAYS FRIENDLY TOWARD STUDENTS. 

KNOWS A LOT ABOUT THE SUBJECT. 

IS NEVER DULL OR BORING* 

EXPECTS A LOT FROM STUDENTS. 

ASKS FOR STUDENTS OPINIONS BEFORE MAKING DECISIONS. 

IS USUALLY CHEERFUL AND OPTIMISTIC. 

IS NOT CONFUSED BY UNEXPECTED QUESTIONS. 

MAKES LEARNING MORE LIKE FUN THAN WORK. 

DOESNT LET STUDENTS GET AWAY WITH ANYTHING 
OFTEN GIVES STUDENTS A CHOICE IN ASSIGNMENTS. 

FRIENDLY AND CHEERFUL. ALOOF AND COLD. 

POISED AND KNOWLEDGEABLE. SELF-CONSCIOUS AND CONFUSED. 

LIVELY AND INTERESTING. DULL AND BORING. 

FIRM AND DEMANDING. EASILY DISTRACTED OR CONFUSED, 

DEMOCRATIC IN PROCEDURE. AUTOCRATIC AND INFLEXIBLE. 

A VERY UNFAVORABLE OPINTON OF THIS TEACHER. 

AN UNFAVORABLE OPINION OF THIS TEACHER. 

A RATHER UNFAVORABLE OPINION OF THIS TEACHER. 

A RATHER FAVORABLE OPINION OF THIS TEACHER. 

A FAVORABLE OPINION OF THIS TEACHER. 

A VERY FAVORABLE OPINION OF THIS TEACHER. 



SLM 


F011 


4423343223 


SLM 


F 0 2 1 


3323144232 


SLM 


F 0 3 2 


4413243231 


SLM 


F041 


4424144341 


SLM 


F 0 5 1 


4334233134 


SLM 


F 06 1 


4413344223 


SLM 


F 07 1 


3434344334 


SLM 


F082 


4434344343 


SLM 


F 09 1 


3322233334 


SLM 


F 1 02 


4434344333 


SLM 


Fill 


4.4.4.4344 3 34 


SLM 


FI 2-1 


3433244322 


SLM 


F 13 2 


4434244443 


SLM 


F142 


443 33443 33 


SLM 


F 1 5 1 


4414344233 


SLM 


F 161 


4434144322 
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SUMMARY OF ST 1 in ENT fv ALU A T T ON OF TEACHTNo 



TEACHER IDENTIFICATION a St.M F161 

RAW SCORES = 375 30l 244 350 231 33« 375 2*3 294 2«1 301 37« ?5s S>' 256 3l8 

THE 16 STUDENTS In THTS CLASS SAID THAT IT TS 
VERY TRUE THAT THIS TEaCHER TS ALWAYS FRIENDLY Town RD STUDENTS. 

VERY TRUE THAT THIS TEACHER KNOWS A LOT ABOUT THE SUBJECT. 

rather false that this teacher is never dull or boring, 
rather true that this teacher expects a lot from students. 

rather false that this teachfr asks for students opinions before making decisions. 
Very true that this TEACHER TS USUALLY CHEERFUL AMn OPTIMISTIC. 

VERY TRUE THAT THIS TEACHER TS NOT CONFUSEO BY UNEx p ECTFO OUESTTOMS. 

rather true that this teacher makes learning more i ike fun than work, 
rather TRUE that THIS TEACHER DOESNT let STUDENTS OFT away with ANYTHING 
RATH'ER TRUE that this TEACHER often GIVES STUDENTS a CHOICE IN ASSIGNMENTS. 

PAIRED ITEMS SUGGEST THAT THTS TEACHER IS 
VFHY FRIENDLY AMU CHEERFUL. 

VF.;HY poised and knowledgeable. 

RATHER LIVELY ANn INTERESTING. 

QUITE FIRM AND DEMANDING. 

rather democratic in procedure. 

IN GENERAL. THIS CLASS HAS A FAVORABLE OPINION OF THIS TEACHER. 




